FAOSTAT 2.3

A revitalisation of the API wrapper of the FAOSTAT API

Author

Sebastian Campbell

Published

April 3, 2023

Abstract

The FAOSTAT package is an important part of the Food and Agriculture Organization (FAO)’s image that is being maintained, but requires a makeover. Here an updated version, 2.3.0, of the package is presented with repaired access to FAO’s API, new functions and preparation for a complete overhaul in 3.0.0. The package has been modernised according to new coding conventions with improved dependencies, documentation and tests. Old useless functions have been pruned and the package is now firmly focused on providing an interface to FAO data to users of R.

Other formats

HTML report | PDF report | Presentation | GitHub

Project background

The motivation for this project came from a Data Mining project from UniLaSalle. It was suggested that we use FAO1 data from their statistical platform FAOstat2. As R was the language of choice, the obvious port of call was the FAOSTAT package3 (Kao, Gheri, and Gesmann 2022), developed by employees at FAO.

  • 1 Food and Agriculture Organization of the United Nations

  • 2 Food and Agriculture Organization Corporate Statistical Database

  • 3 For the purposes of clarity, this document will use the style “FAOSTAT” for the R package and “FAOstat” for the statistical platform

  • However, the FAOSTAT package did not work. It could not download data from the API and could only download bulk data with the entirety of a dataset in one go. For the particular dataset we were interested in, we found that there was a discrepancy between the data in the bulk download and the data on the web platform.4

  • 4 This discrepancy has been fixed as of 2023-03-10

  • Eventually it became necessary to use the same API that the FAOstat website uses to pull data. This method worked and it became clear that it could be used to revitalise the FAOSTAT package and part of an effort to restore it to full functionality.

    FAOstat

    FAOstat is FAO’s web-based statistical platform for the free dissemination of food and agriculture statistics. This data is obtained from questionnaires that FAO distributes throughout the world every year (Food and Agriculture Organization of the United Nations 2019). Some of its data also comes from imputations and models where data is not available, but official country data takes precedence.

    Figure 1: Academic papers referencing FAOSTAT over the last 25 years (Strobel 2018)

    The FAOstat service is a public-facing aspect of FAO, with an overall trend of increasing citations in academic papers year on year with 21400 citations by 2021 (Figure 1).

    Figure 2: FAOstat interface for exploration of country data

    This platform uses a REST API internally to communicate with its database as well as providing a set of zip files with the entirety of certain datasets in order to reduce the load on the database. This REST API allows the website to generate CSVs as well as to allow exploration of the data via interactive graphs (Figure 2).

    FAOSTAT package

    The FAOSTAT package is an API wrapper to pull data from FAOSTAT into a R session. It can also perform small necessary tasks such as country code conversion and coalescing data from different country groups.5

  • 5 For example, China may be just the mainland or may include Taiwan (Chinese Taipei), Hong Kong and Macao

  • History

    The FAOSTAT package was originally developed in 2013 as a tool to source data for the SYB6 project. The yearbooks are yearly summaries of the worldwide state of agriculture for that year. At the time, they were manually typeset and compiled. The new SYB project was to use a combination of LaTeX, knitr and R to automatically pull data from FAOSTAT and other data sources such as the World Bank. This data would be then be transformed and processed to create graphs and tables before finally formatting and typesetting to create a finished product which could then be printed.7. Given that this use case no longer exists, the primary use of this package is for researchers and other R users to read data from FAOstat in a clean way that makes it easier to move to analysis afterwards.

  • 6 Statistical Year Book

  • 7 The author has no insight into the current production of the SYB, but they are still being produced and can be found on the FAO website

  • It is a reasonably popular package in the 86th percentile of all packages on CRAN on 2023-04-01 by downloads. In total, the package has been downloaded over 50 000 times with a peak 121 daily downloads on 2019-05-15. (Li 2023)

    The package was maintained by Michael Kao, the author, from 2013 to 2014. In 2014, it was maintained by Filippo Gheri before passing to Paul Rougieux (the current maintainer) in 2020.

    While it was originally hosted on Github under Michael Kao’s personal account, It is currently hosted on GitLab under Paul Rougieux’s personal account.

    Current state

    The FAOSTAT package has only a shadow of its former functionality. While it has retained the ability to download and process zip files and country code processing functions,8 its capacities are limited by the following issues:

  • 8 For a full description of the status of individual issues, please see the GitLab issue #20 Remove functions linked to defunct uses of FAOSTAT

  • Functionality locked to the Statistical Yearbook

    A number of functions are simply designed to pull in data from other sources such as the World Bank and to process that data into a format easily consumed by the Statistical Yearbook. As the yearbook no longer uses the FAOSTAT package, these functions have no further purpose, serving only to clog up the package and its help files.

    Functionality powered by local files

    Many uses of FAOSTAT require data outside of the data that comes directly from FAOSTAT. The major use case is for code conversions. There are two main code types that require conversion:

    • Country codes
      • FAO: FAO’s internal codes for countries9
      • M49: The UN standard country codes
      • ISO2 & ISO3: 2 and 3 letter country codes
    • Item codes
  • 9 For further details about FAO and how it handles country identification, see FAO’s NOCS database

  • The conversions are not dynamically taken from the API but rather stored in a fixed file. This makes them vulnerable to code or name changes in the future such as the name change of Swaziland to Eswatini in 2018 (United Nations 2018).

    Change of FAOstat API

    The FAOSTAT package is currently configured to access a now-defunct API10 (FAOSTAT3). As a result, it has no methods of retrieving data from the FAOstat platform with the sole exception of the bulk zip downloads which have been since adapted to use the current platform.

  • 10 Originally hosted at faostat3.fao.org

  • Other issues

    The FAOSTAT package is currently maintained by Paul Rougieux who has done an excellent job of keeping the package afloat. However, it’s a small project done in his spare time, so he hasn’t been able to make the time to do a full overhaul. In addition, he isn’t an employee of FAO, but rather of the European Commission. As a face that FAO shows to the world, it seems reasonable that it be placed under its guidance.

    As the package is old, it also has a number of dependencies that are unused or will be after redundant functions are removed. Other dependencies are simply no longer developed or have been superseded by newer packages.

    Project goals

    There are four main goals of this project:

    • Fix up core functions
    • Triage existing functions
    • Characterise relevant aspects of the new API to wrap
    • Transfer maintainership

    Fix up core functions

    The first priority to bring the FAOSTAT package to an acceptable level of functionality is to repair the most basic and most used functions.11

  • 11 The functions below have been paired with a corresponding Gitlab issue that gives more context

  • Functions should not only be repaired but also renamed to have a consistent naming system.

    • getFAO12 - The heart of the package, pulls a custom slice of data from FAOstat
    • FAOsearch13 - Allows a user to find the dataset they’re looking for using the directory in FAOstat
    • translateCountryCode14

    Triage existing functions

    As described in the Current state section, many functions no longer work and need to be removed. This involves a thorough listing of them and their functionality and assessment of their usefulness.

    Characterise relevant aspects of the new API to wrap

    Full overhauling the package is not in scope for FAOSTAT 2.3.0, but rather reserved for the next major version FAOSTAT 3.0.0. 15 This requires the additional step of inspecting the API manually as it doesn’t have complete documentation.

  • 15 For a full set of changes expected for FAOSTAT 3.0.0, see the GitLab milestone page

  • Transfer maintainership

    Uploading a package on CRAN16 requires the permission of the maintainer. At the upload step, an email is sent to the maintainer of the last recorded package version to confirm that the upload was authorised. Only then can the new version proceed to the registration step.

  • 16 Comprehensive R Archive Network, the centralised repository where the majority of R users download R packages

  • Methods

    Talked with existing maintainer Interviewed package author

    Please see issues in milestone: https://gitlab.com/paulrougieux/faostatpackage/-/issues/?sort=created_date&state=closed&milestone_title=2.3.0&first_page_size=20

    • Examining the API

    • Refactor essential functions

      • getFAO -> read_fao
      • FAOsearch -> search_fao
    • Characterise functions GitLab #20

    • Had to repair zip function because of fao search

    • Add caching

    One of the drawbacks of moving from internal data to pulling from the API is the need to read in data every time a lookup table or other piece of reference data is required. This slows down scripts as it’s forced to wait for a reply from the FAOstat server. To mitigate that, caching was implemented. 17 Now instead of pulling every time, the server is polled only once and the response is kept and referred to for subsequent queries.

    • Add column metadata

    Documented using roxygen2 (Wickham et al. 2022), tests written using testthat (Wickham 2011)

    • examples

    Use case

    • Check health of zip data
    • Display availability of certain country data (shiny app)
      • Graph per country
      • Colour by flag

    Future work

    • Release 3.0.0
      • Fully integrating with the new API
    • Publishing a paper in JOSS
    • Publish news in rweekly
    • Publish FAO news alert
    • Move package to FAO repository

    Funding declaration

    This project has been funded by the Food and Agriculture Organization of the United Nations and the author is grateful for their help in reviving it.

    Bibliography

    Food and Agriculture Organization of the United Nations. 2019. Data collection.” https://www.fao.org/statistics/data-collection/en/#jfmulticontent_c728270-2.
    Kao, Michael C J, Filippo Gheri, and Markus Gesmann. 2022. FAOSTAT: Download Data from the FAOSTAT Database.” https://gitlab.com/paulrougieux/faostatpackage https://cran.r-project.org/package=FAOSTAT.
    Li, Peter. 2023. packageRank: Computation and Visualization of Package Download Counts and Percentiles. https://cran.r-project.org/package=packageRank.
    Strobel, Volker. 2018. Pold87/academic-keyword-occurrence: First release,” April. https://doi.org/10.5281/zenodo.1218409.
    United Nations. 2018. Eswatini.” United Nations. https://www.un.org/en/about-us/member-states/eswatini.
    United Nations Statistical Division. 2005. Central Product Classification (CPC) Version 2.1.” Department of Economic; Social Affairs, United Nations. https://unstats.un.org/unsd/classifications/Econ/Download/In Text/CPCv2.1_complete(PDF)_English.pdf.
    Wickham, Hadley. 2011. testthat: Get Started with Testing.” The R Journal 3: 5–10. https://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf.
    Wickham, Hadley, Peter Danenberg, Gábor Csárdi, and Manuel Eugster. 2022. roxygen2: In-Line Documentation for R. https://cran.r-project.org/package=roxygen2.